<html>
<head><meta charset="utf-8"><title>Hang · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html">Hang</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="183823059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823059">(Dec 19 2019 at 07:44)</a>:</h4>
<p>I haven't managed to reproduce it yet, but I just had rustc hang completely while doing a parallel build. I was testing with increasing numbers of threads, so I could plot threads vs user time and see the growth rate. And while doing an 88-way build, rustc just completely hung. I was doing builds of syn, and it hung with two crate builds in progress. One is hung on <code>read</code> of the pipe. The other is hung on a futex.</p>



<a name="183823317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823317">(Dec 19 2019 at 07:49)</a>:</h4>
<p>Looking further, some threads are hung in <code>jobserver::imp::Client::acquire</code> (by way of rayon, <code>rustc_rayon_core::registry::Registry::acquire_thread</code> called from <code>rustc_rayon_core::sleep::Sleep::no_work_found</code>), and others are hung in <code>futex_wait_cancelable</code> via <code>std::sync::condvar::Condvar::wait</code> via <code>rustc_rayon_core::sleep::Sleep::no_work_found</code>.</p>



<a name="183823382"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823382" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823382">(Dec 19 2019 at 07:50)</a>:</h4>
<p>Some backtraces would be useful, but 88 threads would probably make things a bit confusing.</p>



<a name="183823405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823405">(Dec 19 2019 at 07:51)</a>:</h4>
<p>Happy to provide them; my message above was based on looking through backtraces.</p>



<a name="183823714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823714">(Dec 19 2019 at 07:58)</a>:</h4>
<p>This might help:</p>
<div class="codehilite"><pre><span></span>$ grep &#39;#1\&gt;&#39; bt | sed &#39;s/buf=0x[^,]*/buf=(omitted)/&#39; | sort | uniq -c
     86 #1  __GI___libc_read (fd=4, buf=(omitted), nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
      3 #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x563a85f2ced0, cond=0x563a85f2cf00) at pthread_cond_wait.c:502
</pre></div>



<a name="183823721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823721">(Dec 19 2019 at 07:59)</a>:</h4>
<p>(the sed was to handle all the different buffer addresses)</p>



<a name="183823745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823745">(Dec 19 2019 at 07:59)</a>:</h4>
<p>In this 88-way build, that rustc process has 89 rustc threads, of which 86 are blocked in read on the pipe and the other 3 are blocked on <code>pthread_cond_wait</code>.</p>



<a name="183823837"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823837" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823837">(Dec 19 2019 at 08:00)</a>:</h4>
<div class="codehilite"><pre><span></span>     86 #6  0x00007efe03bb8019 in jobserver::Client::acquire_raw (self=&lt;optimized out&gt;) at /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.16/src/lib.rs:363
      3 #6  rustc_rayon_core::sleep::Sleep::sleep (self=&lt;optimized out&gt;, worker_index=&lt;optimized out&gt;, registry=0x563a85f2cf80) at /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc-rayon-core-0.3.0/src/sleep/mod.rs:335
</pre></div>



<a name="183823874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823874">(Dec 19 2019 at 08:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116466">@Zoxc</span> The hung build is still around; what other information can I gather?</p>



<a name="183823901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183823901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183823901">(Dec 19 2019 at 08:01)</a>:</h4>
<p>Are the cond vars all in Rayon?</p>



<a name="183824001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824001">(Dec 19 2019 at 08:03)</a>:</h4>
<p>Maybe just do that command for the 1-7 slots for both processes?</p>



<a name="183824062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824062">(Dec 19 2019 at 08:04)</a>:</h4>
<p>In that process, yes, all the cond vars are in rayon.</p>



<a name="183824064"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824064" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824064">(Dec 19 2019 at 08:04)</a>:</h4>
<p>I just grabbed a backtrace for the other process; checking it now.</p>



<a name="183824069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824069">(Dec 19 2019 at 08:04)</a>:</h4>
<p>...the other process is a lot more interesting.</p>



<a name="183824087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824087">(Dec 19 2019 at 08:06)</a>:</h4>
<div class="codehilite"><pre><span></span>      2 #1  0x00007fdc02953089 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=&lt;optimized out&gt;, ts=...) at /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.6.2/src/thread_parker/linux.rs:111
     72 #1  0x00007fdc029536b9 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=&lt;optimized out&gt;, ts=...) at /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.6.2/src/thread_parker/linux.rs:111
      1 #1  0x00007fdc0295a5a5 in jobserver::imp::Client::acquire (self=0x7fdbf4000b90) at /home/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.16/src/lib.rs:548
     13 #1  __GI___libc_read (fd=4, buf=(omitted), nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
      1 #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x560d777f9e80, cond=0x560d777f9eb0) at pthread_cond_wait.c:502
</pre></div>



<a name="183824239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824239">(Dec 19 2019 at 08:09)</a>:</h4>
<p>Why don't I send you the backtraces of both processes?</p>



<a name="183824309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824309">(Dec 19 2019 at 08:10)</a>:</h4>
<p><a href="/user_uploads/4715/DAVpS_Zm3JaL8FQRZGE8c_jL/bt1.gz" target="_blank" title="bt1.gz">bt1.gz</a></p>



<a name="183824313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824313">(Dec 19 2019 at 08:10)</a>:</h4>
<p><a href="/user_uploads/4715/xJPLH8w5SRG0Sq82ybffNiPD/bt2.gz" target="_blank" title="bt2.gz">bt2.gz</a></p>



<a name="183824342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824342">(Dec 19 2019 at 08:11)</a>:</h4>
<p>Can you filter backtraces to only show symbols so we could get only the unique backtraces?</p>



<a name="183824347"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824347" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824347">(Dec 19 2019 at 08:11)</a>:</h4>
<p>(both obtained by attaching with gdb and running <code>thread apply all bt full</code>)</p>



<a name="183824351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824351">(Dec 19 2019 at 08:11)</a>:</h4>
<p>What do you mean by "only show symbols"? Not addresses?</p>



<a name="183824355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824355">(Dec 19 2019 at 08:11)</a>:</h4>
<p>Or not parameters?</p>



<a name="183824434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824434">(Dec 19 2019 at 08:12)</a>:</h4>
<p>Not parameters, those could vary</p>



<a name="183824651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824651">(Dec 19 2019 at 08:16)</a>:</h4>
<p>I don't know how to hide those in gdb, but I can sed them away easily enough.</p>



<a name="183824940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183824940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183824940">(Dec 19 2019 at 08:22)</a>:</h4>
<p>One moment...</p>



<a name="183825130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825130">(Dec 19 2019 at 08:26)</a>:</h4>
<p>OK, I managed to get the <em>unique</em> backtraces.</p>



<a name="183825294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825294">(Dec 19 2019 at 08:29)</a>:</h4>
<p>In the less interesting process, there are only two unique backtraces, both as you'd expect.</p>



<a name="183825375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825375">(Dec 19 2019 at 08:30)</a>:</h4>
<p><a href="/user_uploads/4715/9yN7JNlznpjIl8ZEJgJN-EvR/bt1-uniq" target="_blank" title="bt1-uniq">bt1-uniq</a></p>



<a name="183825474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825474">(Dec 19 2019 at 08:32)</a>:</h4>
<p>The more interesting process has no two backtraces identical.</p>



<a name="183825732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825732">(Dec 19 2019 at 08:37)</a>:</h4>
<p>It kind of looks like the first processes is waiting for a jobserver token. Maybe the other process grabbed all 88 tokens? Or some tokens are leaking somehow</p>



<a name="183825793"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825793" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825793">(Dec 19 2019 at 08:38)</a>:</h4>
<p>It would be nice if you could reliable recreate it, so we could see if it would happen with no jobserver used</p>



<a name="183825824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825824">(Dec 19 2019 at 08:39)</a>:</h4>
<p>I tried many times and didn't recreate it. That's why I'm keeping it around.</p>



<a name="183825835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825835">(Dec 19 2019 at 08:39)</a>:</h4>
<p>It does look like the other process is in the midst of all the work.</p>



<a name="183825924"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183825924" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183825924">(Dec 19 2019 at 08:41)</a>:</h4>
<p>I managed to common them up a bit more.</p>



<a name="183826292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826292">(Dec 19 2019 at 08:47)</a>:</h4>
<p>Well thread 89 looks interesting already since it's in monomorphization and not Rayon</p>



<a name="183826394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826394">(Dec 19 2019 at 08:49)</a>:</h4>
<p>So it's probably some deadlock there</p>



<a name="183826515"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826515" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826515">(Dec 19 2019 at 08:51)</a>:</h4>
<p>(Well, it is in Rayon, but it's doing actual parallel work, and then taking a lock.)</p>



<a name="183826618"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826618" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826618">(Dec 19 2019 at 08:53)</a>:</h4>
<p>Everything is in Rayon thread, but the lock it's stuck on is not a Rayon lock</p>



<a name="183826626"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826626" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826626">(Dec 19 2019 at 08:53)</a>:</h4>
<p><a href="/user_uploads/4715/QoDGulin5WZnKzz_RlSgowrg/bt2-uniq" target="_blank" title="bt2-uniq">bt2-uniq</a></p>



<a name="183826640"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826640" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826640">(Dec 19 2019 at 08:53)</a>:</h4>
<p>That's the unique backtraces in the second process, for frames 0-13.</p>



<a name="183826692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826692">(Dec 19 2019 at 08:54)</a>:</h4>
<p>That shows which ones are in monomorphization without getting into the non-unique frames.</p>



<a name="183826808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826808">(Dec 19 2019 at 08:56)</a>:</h4>
<p>I need to go to bed. I'll leave the processes running until tomorrow morning. If you think of any other information you want from them, let me know.</p>



<a name="183826814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826814">(Dec 19 2019 at 08:56)</a>:</h4>
<p>Oh, it's only 72? I was expecting more</p>



<a name="183826829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826829">(Dec 19 2019 at 08:57)</a>:</h4>
<p>I think this should suffice to track it down, unless there's multiple bugs</p>



<a name="183826924"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826924" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826924">(Dec 19 2019 at 08:59)</a>:</h4>
<p>This was run with <code>-Zthreads=88</code>.</p>



<a name="183826988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183826988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183826988">(Dec 19 2019 at 09:00)</a>:</h4>
<p>Note that since I only have 72 cores, perhaps only 72 were getting to run?</p>



<a name="183827000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183827000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183827000">(Dec 19 2019 at 09:00)</a>:</h4>
<p>Thanks for digging into this!</p>



<a name="183827456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183827456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183827456">(Dec 19 2019 at 09:07)</a>:</h4>
<p>I guess cargo would limit to 72, unless you did -j88 too?</p>



<a name="183827463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183827463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183827463">(Dec 19 2019 at 09:07)</a>:</h4>
<p>72 would make sense then</p>



<a name="183830356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183830356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183830356">(Dec 19 2019 at 09:50)</a>:</h4>
<p>The problem is that <code>record_accesses</code> calls <code>is_inlining_candidate</code> inside the iterator, which is enumerated inside the lock. That should instead be called when appending the item to <code>neighbors</code> in <code>collect_items_rec</code>.</p>



<a name="183855958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183855958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183855958">(Dec 19 2019 at 15:34)</a>:</h4>
<p>That makes sense.</p>



<a name="183855966"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang/near/183855966" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Hang.html#183855966">(Dec 19 2019 at 15:34)</a>:</h4>
<p>Nice catch!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>